Neural network classification using Shannon's entropy

The last years have witnessed an increasing attention to entropy-based criteria in adaptive systems. Several principles were proposed based on the maximization or minimization of entropic cost functions. We propose a new type of neural network classifiers with multilayer perceptron (MLP) architecture, but where the usual mean square error minimization principle is substituted by the minimization of Shannon’s entropy of the differences between the MLP’s output and the desired target. The backpropagation algorithm is optimized with a variable learning rate and tested in five well known datasets. The results show a very good performance of MLPs trained with Shannon’s entropy when compared with the mean square error and cros-entropy criteria.